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ABSTRACT  (U) 

(U)  The  fusion  of  visual  and  infrared  sensor  imagery  for  the  detection  of 
concealed  weapons  is  demonstrated  using  several  techniques.  The  fusion 
algorithms  used  are  wavelet  based  fusion  and  Fuzzy  Logic  Approach  (FLA) 
fusion.  The  FLA  is  presented  as  one  of  several  possible  methods  for  combining 
images  from  different  sensors  for  achieving  an  image  that  displays  more 
information  than  either  image  separately.  Near  infrared  filters  are  used  along  with 
camcorders  to  capture  images  for  later  fusion.  Metrics  are  suggested  that  could 
rate  the  fidelity  of  the  fused  images,  such  as,  a  textured  clutter  metric  and  entropy. 


INTRODUCTION  (U) 

(U)  Recent  terrorist  incited  events,  both  at  home  and  abroad,  have  created  the  need  for  the 
development  of  improved  methods  for  the  detection  of  concealed  weapons  to  improve  homeland 
security.  The  locations  that  require  improved  concealed  weapon  detection  capabilities  are  many; 
airport  gates,  building  entrances,  borders,  urban  environments  etc.  Concealed  weapon  detection 
ideally  should  be  reliable  and  fast,  however,  as  with  any  imaging  and  detection  technology,  there 
are  tradeoffs  to  consider  that  involve  technical  as  well  as  social  issues.  Portability  and  imaging 
speed  place  constraints  on  the  size  and  resolution  of  sensors  and  the  computer  systems  that  are 
used  to  control  the  imaging  devices.  In  addition  to  size,  public  exposure  to  ionizing  radiation  is 
another  issue  that  must  be  considered  in  technology  choices  that  try  to  balance  better  concealed 
weapon  detection  rates  with  increased  public  safety  and  confidence. 

(U)  The  authors  have  tested  or  obtained  example  imagery  from  sensors  in  several  wavelength 
regimes  for  detection  of  concealed  weapons  and  describe  in  this  paper  the  sensor  fusion  of  near 
IR  and  visible,  near  IR  and  far  IR,  mmwave  and  visual  band  imagery.  Combined  with  the  afore 
mentioned  images,  the  authors  have  tested  several  computational  methods  to  combine  and 
enhance  the  images  from  the  sensors,  such  as  wavelet  edge  processing,  frizzy  logic  fusion, 
Gaussian  Laplacian  pyramid  fusion,  and  others. 
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(UMethod 

(U)  The  sources  of  imagery  for  sensor  fusion  were  from  sensors  in  the  Visual  Perception 
Laboratory  at  TACOM  and  raw  imagery  supplied  from  other  laboratories.  The  images  were  then 
combined  by  using  either  the  MATLAB  Fuzzy  Inference  System  (FIS)  and  the  FuseTool 
interface.  In  his  book  Multi-Sensor  Fusion  \  Brooks  points  out  that  Fuzzy  logic  is  a  technology 
that  shows  “promise  for  use  with  sensor  problems.”  He  goes  on  to  mention,  however,  that 
because  of  the  numerous  forms  of  membership  functions,  methods  of  recombination,  etc.,  it  is 
difficult  to  know  exactly  which  implementation  is  best  suited  for  use  in  sensor  fusion 
technology.  The  fuzzy  logic  approach  to  image  fusion  algorithm  is  described  as  one  option  for 
fusion  below.  Future  papers  will  define  and  compare  other  methods  and  algorithms  for  image 
fusion. 

(U)  Fuzzy  Logic  Based  Fusion 

(U)  A  great  deal  of  interest  has  been  shown  in  the  Fuzzy  Logic  Approach  (FLA)  during  the  last 
three  decades  for  numerous  technical  areas  A  strong  point  of  the  FLA  is  that  it  permits 

the  encoding  of  expert  knowledge  directly  and  easily  using  rules  with  linguistic  labels.  A  weak 
point  is  that  it  usually  takes  some  time  to  design  and  tune  the  membership  functions  that 
quantitatively  define  the  parameters  of  interest.  To  enable  a  system  or  process  to  deal  with 
system  level  uncertainties,  researchers  have  incorporated  the  concept  of  fuzzy  logic  into  many 
control  systems.  It  has  been  found  that  artificial  neural  network  learning  techniques  can  automate 
this  process  and  substantially  reduce  development  time  while  improving  performance  In  this 
paper,  one  of  the  techniques  that  the  authors  demonstrate  sensor  fusion  for  is  the  purpose  of 
concealed  weapon  detection  using  the  FLA. 

(U)  The  basic  algorithm  for  pixel  level  image  fusion  using  the  fuzzy  logic  approach  is; 

•  Read  first  image  in  variable  il  and  find  its  size  (rows:  zl,  columns:  si). 

•  Read  second  image  in  variable  i2  and  find  its  size  (rows:  z2,  columns:  s2). 

•  Variables  il  and  i2  are  images  in  matrix  form  where  each  pixel  value  is  in  the  range  from 
0-255.  Use  gray  color  map. 

•  Compare  rows  and  columns  of  both  input  images,  starting  from  the  upper  left.  If  the  two 
images  are  not  of  the  same  size,  select  the  portions  which  are  of  same  size. 

•  Convert  the  images  in  column  form  which  has  C=  zl*sl  entries. 

•  Make  a  Fuzzy  Inference  System  file  which  has  two  input  images,  (See  Fig.’s  1  and  2). 

•  Decide  the  number  and  type  of  membership  functions  for  both  the  input  images  by 
adjusting  the  membership  functions.  Input  images  in  antecedent  are  resolved  to  a  degree 
of  membership  between  0  to  255. 

•  Make  rules  for  two  input  images  which  resolves  the  two  antecedents  to  a  single  number 
from  0  to  255. 

•  For  num=l  to  C  in  steps  of  one,  apply  fuzzification  using  the  rules  developed  above  on 
the  corresponding  pixel  values  of  the  input  images  which  gives  a  fuzzy  set  represented  by 
a  membership  function  and  results  in  output  image  in  column  format. 

•  Convert  the  column  form  to  matrix  form  and  display  the  fused  image. 
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i  FIS  Editor:  fuse  ei  chdnae2  ET 


Fig.  1  (U)  Mamdani  FLA  FIS 


Fig.  2:  (U)  Mamdani  MF’s 


lU)  Wavelet  Processing  of  Fused  Images 

(U)  The  computational  method  used  by  the  authors  to  enhance  the  fused  images  is  the 
application  of  wavelet  processing.  Wavelets  have  been  used  by  the  signal  and  image  processing 
communities  for  several  years  now.  A  brief  review  of  wavelets  for  image  processing  will  be 
given  below  and  then  some  examples  of  wavelet  enhanced  fused  imagers  will  be  shown. 

(U)  A  fairly  old  method  of  computing  a  local  spectrum  is  to  apply  the  Fourier  Transform 
(FT)  to  one  specific  piece  of  the  signal  at  a  time.  This  is  the  idea  behind  what  is  called  the 
Windowed  Fourier  Transform  (WFT).  Basically,  the  implementation  involves  using  a 
rectangular  window  to  isolate  a  portion  of  the  signal  of  interest,  which  is  then  Fourier 
transformed.  As  the  window  slides  along  to  different  positions,  the  WFT  gives  the  spectra  at 
these  positions.  This  kind  of  analysis  has  a  fundamental  problem  however,  whose  mathematics 
is  similar  to  the  Heisenberg  Uncertainty  Principle  in  Quantum  Mechanics  (QM).  Multiplying  the 
signal  by  a  window  function  results  in  convolving  or  mixing  the  signal  spectrum  with  the 
spectrum  of  the  window.  Add  this  to  the  fact  that  as  the  window  gets  smaller,  its  spectrum  gets 
wider,  and  we  have  the  basic  dilemma  of  localized  spectra:  the  better  we  determine  the  position 
of  the  signal,  the  poorer  we  localize  the  spectrum.  This  is  analogous  to  the  case  in  QM  where 
increased  precision  in  a  description  of  say  the  momentum  of  an  electron  reduces  the  precision 
available  of  the  position  of  that  electron.  Very  accurate  determinations  can  be  made  or 
computed,  but  both  are  not  available  to  an  unlimited  degree  of  precision.  Correspondingly,  there 
is  a  fundamental  physical  limit  to  the  degree  of  precision  of  the  frequency  content  of  a  signal  at  a 
particular  position  [7,8]. 

(U)  In  1946  Dennis  Gabor  [8]  introduced  a  version  of  the  WFT  that  reduced  this 
uncertainty  somewhat.  The  Gabor  transform  uses  a  Gaussian  profile  for  the  window  since  the 
Gaussian  is  the  function  that  minimizes  this  uncertainty.  However,  the  underlying  idea  of 
localizing  a  spectrum  of  a  signal  by  windowing  the  signal  needs  to  be  reconsidered.  Obviously, 
care  must  be  taken  in  the  selection  of  the  signal.  Careful  attention  to  the  placement  of  the 
window  however  is  not  an  easy  task  for  realistic  time-varying  signals.  We  are  in  fact  trying  to 
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do  two  different  things  at  once.  Frequency  is  a  measure  of  cycles  per  unit  time  or  signal  length. 
So  that  high  frequency  oscillations  take  much  less  signal  length  or  time  than  do  low  frequency 
oscillations.  High  frequencies  can  be  well  localized  in  the  overall  signal  with  a  short  window, 
but  low  frequency  localization  requires  a  long  window.  The  wavelet  transform  takes  an 
approach  that  permits  the  window  size  to  scale  to  the  particular  frequency  components  being 
analyzed. 

(U)  Wavelets  are  generally  speaking,  functions  that  meet  certain  requirements.  The  name 
wavelet  originates  from  the  requirement  of  integrating  to  zero  by  oscillating  about  the  x-axis  and 
being  well  localized  In  fact,  there  are  many  kinds  of  wavelets.  There  are  smooth  wavelets, 
wavelets  with  simple  mathematical  expressions,  wavelets  that  are  associated  with  fdters,  etc. 
The  simplest  wavelet  is  the  Haar  Wavelet. 

(U)  There  are  many  important  characteristics  of  wavelets  that  make  them  more  flexible 
than  Fourier  analysis.  Fourier  basis  functions  are  localized  in  frequency  but  not  in  time.  Small 
frequency  changes  in  a  FT  cause  changes  ever5where  in  the  time  domain.  Wavelets  are  local  in 
both  frequency  and  scale  by  the  use  of  dilations  and  in  time  by  the  use  of  translations.  This 
ability  for  localization  is  useful  in  many  applications.  Another  advantage  of  wavelets  is  coding 
efficiency.  Many  classes  of  functions  can  be  represented  by  wavelets  in  a  compact  way. 
Functions  that  have  discontinuities  and  sharp  spikes  take  fewer  basis  functions  to  reach  a  similar 
approximation. 

(U)  Noisiness  extends  to  the  realm  of  image  data  sets.  Noisy  data  sets  can  be  cleaned  up 
by  using  wavelets.  Typically  the  speed  of  wavelets  is  also  much  faster  than  the  fastest  Fast 
Fourier  Transform  (FFT).  The  data  is  basically  encoded  into  the  coefficients  of  the  wavelets. 
The  computational  complexity  of  the  FFT  is  of  the  order  of  (nlog2(n))  whereas  for  most  wavelets 
the  order  of  complexity  is  of  the  order  n.  Many  data  operations,  such  as  multiresolution  signal 
processing  can  be  done  by  processing  the  corresponding  wavelet  coefficients. 

The  basic  flow  of  processing  in  wavelet  analysis  is  shown  below  in  Fig.3: 


Fig.  3:  (U)  Wavelet  processing  flow  diagram 

(U)  How  wavelets  work  is  best  illustrated  by  way  of  an  example.  The  simplest  of  all 
wavelets  is  the  Haar  wavelet  The  Haar  wavelet,  y/{x),  is  a  step  function  that  assumes  the 
values  1  and  -1.  The  Haar  wavelet  is  more  than  80  years  old  and  has  been  used  for  various 
applications.  It  can  be  shown  that  any  continuous  function  can  be  approximated  by  Haar 
functions.  Dilations  and  translations  of  the  function^(x), 

y/ ji^(x)  =  const  y/{2^ X  -  k)  (1) 
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(U)  define  an  orthogonal  basis  in  the  space  of  all  square  integrable  functions,  L  (R).  This  means 
that  any  element  of  L^(R)  can  be  represented  as  a  linear  combination  of  these  basis  functions. 
The  orthogonality  of  the  wavelets  pair  is  checked  by  the  following, 

fk'  ~  ^ jj  ^  k'k',  (2) 


(U)  whenever  j=j'  and  k=k'  is  not  satisfied  simultaneously.  The  constant  that  makes  the 
orthogonal  basis  orthonormal  is  2^^^  . 

tut  IMAGES 

(U)  The  figures  below  are  samples  of  both  the  raw  input  images  from  several  sensors  and  the 
fused  images.  Fig.  4  is  the  visible  image  of  a  black  sweater  suspended  over  a  board.  Beneath  the 
sweater  is  a  pair  of  metal  office  scissors  that  is  not  visible  to  the  naked  eye.  In  Fig.  5,  the  picture 
is  a  long  wave  infrared  image  of  the  scene.  Since  the  scissors  are  being  warmed  by  the  sun 
outside,  the  infrared  imager  can  form  a  reasonable  good  image  based  on  the  temperature 
difference  between  the  sweater  and  the  pair  of  scissors.  The  image  in  Fig.  6  was  obtained  using 
a  near  infrared  filter  with  a  commercial  off-the-shelf  (OTS)  camcorder  with  night-shot 
capability.  The  outline  and  partial  image  of  the  pair  of  scissors  can  be  seen  through  the  sweater; 
in  fact,  both  the  handles  and  blades  are  clearly  visible.  Fig.  7  is  a  wavelet  fused  and  enhanced 
image  of  Fig.  6.  In  this  case,  the  enhanced  image  shows  the  outline  of  where  the  scissors  are. 


Fig.  6:(U)  Near  IR  image  of  scissors 


Fig.  5:(U)  Far  IR  image  of  scissors 


Fig.  7(U):  Wavelet  Fused  near  and  far  IR 
and  enhanced  image 
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Fig.  9:(U)  near  IR  image  of  bar  pattern 
under  pants 


Fig.  10:(U) visual  image" 


Fig.  1 1  :(U)passive  mmwave  "  Fig.  12:(U)FLA  fused 


(U)  Fig.’s  8  and  9  are  the  visible  and  near  IR  image  of  thick  cotton  pants  that  have  a  bar  code 
paper  under  one  of  the  pant  legs.  The  camcorder  with  the  near  IR  fdter  clearly  detects  the  bar 
pattern  under  the  pant  leg.  Future  work  will  demonstrate  the  detectability  of  various  concealed 
weapons.  One  of  the  benefits  of  using  multi-band  IR  is  that  the  imaging  process  is  completely 
passive,  no  ionizing  radiation  is  used  or  directed  unto  the  people  being  scanned.  The  drawback 
with  such  a  technique  is  that  there  is  a  loss  in  resolution  and  additional  computer  processing,  be 
it  some  kind  of  edge  enhancement  or  image  processing  such  as  sensor  fusion  must  be  used  in 
order  to  detect  a  weapon,  especially  if  thick  clothing  is  covering  the  weapon.  Fig.’s  10,  11  and 
12,  show  the  imaging  of  a  concealed  weapon  with  passive  millimeter- wave  (mmwave).  "  Passive 
millimeter- wave  forms  an  image  from  the  radiation  emitted  from  the  body  by  virtue  of  it’s 
temperature,  no  additional  incident  radiation  is  needed.  The  image  in  Fig.  12  is  fused  using  a 
Fuzzy  Logic  Approach  (FLA)  fusion  algorithm  to  segment  the  location  of  the  handgun. 

(U)  How  the  Near  IR  filter  works 

(U)  The  part  of  the  electromagnetic  spectrum  that  is  visible  to  the  naked  eye  lies  between  the 
wavelengths  of  400nm  to  approximately  760nm.  (lnm=10'^m)  Infrared  rays  have  longer 
wavelengths  than  the  visible  band  wavelengths,  and  range  from  760  nm  to  3000  nm,  and  are  also 
sometimes  called  the  short-wave  IR  or  SWIR  bands  of  the  IR,  or  near-IR.  The  near-IR  imager 
works  by  using  a  filter  with  a  camcorder."  The  near  IR  electromagnetic  waves  (EM)  incident 
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on  the  subject  (2)  from  the  source  (1)  penetrate  the  clothing  because  of  their  wavelength,  are 
absorbed  and  re-emitted  from  the  person,  and  then  again  pass  out  through  the  clothing  (4)  on  the 
way  to  the  camera  and  can  be  used  to  form  an  image  on  the  CCD  chip  (5).  The  filter  (6)  used 
with  the  camcorder  passes  only  the  near  IR  EM  radiation.  Since  most  CCD  chips  used  in  OTS 
camcorders  are  sensitive  to  this  part  of  the  spectrum,  an  image  can  be  formed.  In  some  cases  the 
image  is  good  enough  to  see  objects  under  layers  of  clothing,  and  sometimes  additional  image 
processing  is  needed  to  make  the  concealed  object  visible. 


lU)  Analysis 

An  entropy  and  texture  based  clutter  metric  were  run  over  the  images  to  see  if  there  was  a 
correlation  of  the  visual  quality  of  the  fused  imagery  with  the  metrics.  The  entropy  of  an  image  is 
a  measure  of  the  information  content,  in  terms  of  gray  scale  levels  and  is  also  related  to  the 
texture  of  the  image.  The  maximum  value  the  entropy  metric  can  take  on  is  eight  and  the 
minimum  is  zero.  The  equation  used  for  the  calculation  of  the  entropy  [10]  is  shown  below. 


H=  pig) 

g=0 

where  p(g)is  the  probability  of  gray  value  g,  and  the  range  of  g  is  [0,...,L-1]. 


MMW  Weapon  Images  Scissor  Images 


Sensor 

Entropy  Text  Glut 

Sensor 

Entropy  Text  Glut. 

Visible 

6.653 

59.17 

Visible 

7.3932 

44.02 

Mmwave  Radar 

6.7709 

61.64 

IR 

6.2709 

16.42 

Fused 

7.1993 

65.39 

Fused  PCA 

7.4658 

66.75 

Fused  PCA  edge 

6.5238 

89.82 

Table  1 :  (U)  Entropy  and  clutter  metric  values  for  sensor  images 
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(U)  Conclusions 

(U)  In  summary,  the  authors  have  shown  some  of  the  sensor  combinations  and  algorithms  that 
can  be  combined  to  detect  concealed  weapons.  Which  sensors  are  used  is  going  to  be  determined 
also  by  factors  such  as  cost,  perceived  risk  by  the  public,  and  the  size  of  the  device.  Using 
passive  infrared  and  or  millimeter-wave  multiband  imagery  with  sensor  fusion  and  edge 
enhancement  it  is  possible,  in  some  circumstances  to  detect  a  weapon  concealed  under  light 
clothing.  When  imaging  through  thick  layers  and  a  high  degree  of  resolution  is  required,  the  best 
sensor  may  be  active  millimeter-wave  imagers,  however,  this  capability  brings  with  it  greater 
cost.  The  authors  have  shown  some  of  the  imaging  possibilities  using  passive  infrared  imagery 
and  millimeter-wave  wave  images.  Future  research  could  show  how  the  passive  infrared  sensors 
used  with  image  fusion  and  processing  algorithms  can  detect  concealed  weapons  such  as 
handguns. 

(U)  References 

[1  .].(U)  R.R.  Brooks,  S.S.  Iyengar,  Multi-Sensor  Fusion:  Fundamentals  and  Applications 
with  Software,  Prentice  Hall  PTR,  Upper  Saddle  River,  N.  J.,  p.  167,  1997 

[2] .  (U)  L.  Zadeh,  "Fuzzy  Sets",  Information  and  Control",  8,  pp.  338-353,  1965. 

[3] .  (U)  E.  Mamdani  and  S.  Assilian,  "Applications  of  fiizzy  algorithms  for  control  of  simple 
dynamic  plant",  Proc.  Inst.  Elec.  Eng.,  Vol.  121,  pp.  1585-1588,  1974. 

[4] .  (U)  T.  Munakata,  and  Y.  Jani,  "Fuzzy  Systems:  An  Overview",  Commun.,  ACM,  Vol.  37, 
No.  3,pp.  69-76,  Mar.  1994. 

[5] .  (U)  E.  Cox,  The  Fuzzy  Systems  Handbook:  A  Practitioner's  Guide  to  Building,  Using,  and 
Maintaining  Fuzzy  Systems,  AP  Professional,  1994. 

[6] .  (U)  D.  G.  Schwartz,  G.  J.  Klir,  H.  W.  Lewis,  and  Y.  Ezawa,  "Applications  of  Fuzzy  Sets  and 
Approximate  Reasoning",  IEEE  Proc.,  Vol.  82,  No.  4,  pp.  482-498,  Apr.  1994. 

[7] .  (U)  Meitzler,  T.  and  Singh,  H.,  “Extension  of  2-D  wavelet  transforms  of  images  to  3-D 
transforms  and  their  applications,”  IEEE  Transactions  on  Aerospace  and  Electronic  Systems, 

Vol.  34,  (3),  p.  963,  July,  1998. 

[8] .  (U)  Freeman,  M.O.,  “Wavelets-signal  representations  with  important  advantages,”  Optics 
abd  Photonic  News,  4,  8  (Aug.  1993),  pp.  8-14. 

[9] .  (U)  Vidakovic,  B.,  and  Muller,  P.,  Wavelets  for  kids,  Durham,  N.C.,:  Duke  University 

[10] .  (U)  Wang,Y.,  Lohmann,  B.,  “Multisensor  Image  Fusion:  Concept,  Method  and 
Applications,”  Internet  source. 

[11] .  (U)  Wikner,  D.,  Army  Research  Laboratory,  Millimeter-wave  Branch,  dawik@arl.army.mil 

[12] .  (U)  Kaya-Optics  web  site,  http://www.kava-optics.com/products/overview.shtmk  2003. 


UNCLASSIFIED 


